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I. INTRODUCTION. 



Testing hardware systems to determine whether they satisfy operational 
specifications is frequently an expensive and time-consuming task. In 
some circumstances, it is possible to decrease the consumption of resources 
in testing by using sequential test procedures. A sequential test requires 
tnat limits be placed upon the desired precision of the test (probabilities 
of errors), and that the alternatives under test (hypotheses) be clearly 
specified. Such a test with two hypotheses is conducted by taking one 
observation (test value) at a time and, after each observation, making 
one of the decisions 

(a) terminate and reject the null hypothesis; 

(b) terminate and reject the alternate hypothesis; 

(c) take another observation. 

Since a sequential procedure requires that a decision (a - c above) 
be made after each observation, it is necessary to be able to obtain the 
observation value before another test is run (assuming it does not 
terminate at that point). In the case of testing a bombing system where 
several test runs may be scheduled in a relatively short span of time, this 
means that the measure being used in the test must essentially be obtainable 
in real time, without appreciable processing delays. Similarly, some 
"field computation", again without long delays, may be required in order to 
reach one of the decisions a - c. 

In what follows we discuss two sequential test procedures for testing 
whether a specified CEP (or less) has been achieved by a weapon system 
under given test conditions. The alternate hypothesis is that the CEP 



is a second, larger, value or more. These values are obtained from 
contract specifications and operational considerations. One of the 
procedures is based on the assumption that bomb impacts follow a 
circular normal distribution, so that radial miss distances are 
Rayleigh distributed. Even though this parametric model is frequently 
used in practice, its adequacy depends upon how well the Rayleigh fits 
the real situation. There is mounting evidence [1] that the Rayleigh 
model does not fit actual drop data in the upper tail of the distri- 
bution. Hence estimates (such as CEP = 1.1774 a) and tests based on 
this model may not perform well, or even acceptably. In addition, the 
amount of processing required to obtain each miss distance may involve 
an intolerable delay before a decision is reached about taking another 
observation. For these reasons, we give below, only a mathematical 
sketch of the corresponding sequential test, which we call the "Rayleigh 
sequential test." 

We describe the second test procedure as "nonparametric," in that it essen 
tially does not depend on assumptions about the distribution of bomb impact 
points about the target. This procedure utilizes the observed proportion 
of impacts that fall within the specified CEP. It is assumed that 
judgement about whether a given bomb fell within the specified CEP can be 
made without significant processing delay, so that it will be known 
shortly after a given bomb impact is observed, whether another test run 
is required. This sequential procedure is mathematically equivalent 
to a well known procedure frequently used in reliability testing and 
quality assurance applications; in what follows we call it the "sample 
proportion test." 
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II. DESCRIPTION OF TESTS. 



We use the Sequential Probability Ratio Test (SPRT) developed by 
Wald [2], The test may be described in terms of a random walk 
and bounds B < A, such that the test continues so long as tt^ 
remains between B and A. The statistic 7t n is based on the likelihood 
ratio, which in turn depends upon the statistical model (in our case, 
Rayleigh or binomial) and the CEP values specified under the hypotheses. 

We shall employ the usual approximations to the stopping bounds B and 
A, 



wnere a and 3 are the type I and type II error rates, respectively. 
There is strong evidence [3] that these approximations are quite adequate, 
except possibly for very small a and e . 

A. Rayleigh Test . It is convenient to state the test in terms of 

2 2 
CEP and squared radial miss distances, R . Since the original quantities 

are nonnegative, this test is equivalent to what would be obtained with 

CEP and R. . Now assume the Rayleigh model for R, so that 

2 

R 2 ~ F R2 (y) = l-e“ y/2 ° ; y > 0 , (1) 

2 

an exponential distribution with parameter A = l/2o . Hypothesizing 

p 

CEP = C is equivalent to setting A = a/Z c (say). 

Let Cq denote the squared CEP specified and C^ denote the 
squared CEP alternative, where (C Q ,C-|) is an i ndjfference zone. Presumably 
we are indifferent to which hypothesis is rejected when in fact the true 
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2 

CEP squared is between C Q and . In order to test CEP <; C Q against 
CEP 2 :> C^ , we test the simple hypotheses 

H q : CEP 2 = C Q vs H 1 : CEP 2 = C ] (2) 



The SPRT continues so long as 



n 

B < 7T 

i=l 



f(y i ; c 1 ) 

fTy~ry 



< A , 



(3) 



where f(y i ; Cj) , 
is 



the density corresponding to (1) with — ^ = a/C. , 

2 a J 



f ^ y i ’ C j^ = c~ exp{_ a i /C j^ * 



(4) 



Taking logs in expression (3), with the explicit expression (4) for f, 
the test may be described as follows: 

SEQUENTIAL RAYLEIGH TEST : In order to sequentially test hypotheses (2) with 

approximate size a and power 1-3 when sampling from density (4), con- 

tinue taking observations R so long as their sum (the sum of the first 

n observations) falls between the acceptance numbers a n and the rejection 

numbers r , where 
n 



a n = <" ,n ci + 1n feAc^- C7 ] “ • 



'0 
C 1 

r = (n 1 rbr- + 1 
n C Q 



cv ] “ • 



Terminate and accept H n whenever \ R <; a ; terminate and reject H r 



n 2 

whenever T R ^ r 
r=l r n 



r=l 
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The operating characteristic L of this test is approximately 



L(C) = P[Test accepts H n | CEP 2 = C] « (A h(C) - l)/(A h(C) - B h(C) ) , 



where h(C) is any (positive) solution to the equation 



,h(C ) 



1 



h(C) [J- - J-] 
L 1 L o 



Let N denote the (random) sample size at termination. The average 

2 

sample size required for termination when CEP = C is approximately 



E(N I CEP'" = C) 



[1 -L(C )]1 n ^ + L(C)ln 1 | 5 - 
ln ^ + C[J- - M 



B. Sample Proportion Test . In order to test the hypotheses (2) 

without making strong assumptions about the distribution of impact points 

about the target, one may transform each data point by an indicator function 

which gives "hit-miss" data. Suppose, then, that is 1 if ^ /C^ ; 

X.j = 0 otherwise. Assuming the miss distances , R^, ... are a random 

sample from an (unknown) continuous distribution, X^ , X^, ... are 1 1 D 

Bernoulli random variables under Hq and H a . For these transformed 

2 1 

variables the null hypothesis Hq : CEP <; Cq is equivalent to Hq : p :> 1/2 , 

where p is the "success" parameter, p = P[X = 1], We also need to 

determine limits on p under H : CEP 2 £ C, . This requires a model which 

cl I 
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For this 



relates P [ ^ /Cq ] to C-j , through the distribution of . 
purpose, we assume a Rayleigh formula. However, this is not a strong 
assumption nere (in contrast to the parametric model discussed above), 
because we do not rely on the formula for values far from the hypothesized 
medians of the distributions. It has been noted above that the Rayleigh model 
gives adequate fit for these values; it is the tai 1 behavior of the Ray- 
leigh that seems to invalidate its use for many CEP estimation and testing 
problems. Since we are using the Rayleigh formula for moderate values, 
and in fact using it only to assess P[X.j = 1 | CEP = C-j], we feel this 
is not a serious assumption and that it is not misleading to refer to the 

present test as a "nonparametric" test. Under this assumption and with 

2 

the transformed variables X., the alternate hypothesis H : CEP ^ C, 

1 d I 

i 

is equivalent to ; p ^ p-| , where 

P] = P[X. = 1 | CEP 2 = C,] = P[R 2 C Q | CEP 2 = C^ 



= 1 - exp 



K a ) 



= 1-2 



-V C 1 



The likelihood ratio in this case is 



n 

Z X. 
1-1 1 



n f(X i ; Pl } / Pi 1 fon n n n 

^ f(X. ; 1/2) " (jPp ~) C2(1 " P 1 )] ’ 



which remains between B and A so long as £ X. is between 

i=l 1 



, n = (,„ 1 i E+ „( c c p.,) B )/ 4 c ° /c >. ,) 



(4) 
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and 



r n ■ (’" Hr + - ’) 



/ In 2 



V C 1 



1J. 



the "acceptance number" and "rejection number," respectively. Note 
that the test is dependent upon and C-j only through their ratio, 

C (/ C l ' 

We can now state our "nonparametric" sequential CEP test as follows: 
SAMPLE PROPORTION TEST . In order to sequentially test hypotheses 
(2) with approximate size a and power 1-3 , make independent observa- 
tions of whether each bomb radial misdistance is within (so X.. = 1 ) 

or is not (so = 0). Continue taking observations X^ so long as 

a n < -^ ^i < r n w * iere a n anc * r n are 9i ven i n (4). Terminate and accept 

^ ^ n n 

Hq whenever . 7^ X^ ^ a^ ; terminate and reject Hq whenever E^ X^ r n . 

The operating characteristic L of this test is approximately 
L(C) - P(test accepts H Q | CEP 2 = C) « (A h(C) - l)/(A h ^ C) - B h(C) ) , 
where h(C) is any positive solution to the equation 



C = - C Q a / ln(l-J) , 



wnere 



, . 2 h(C)( 2 - C 0 /C l) h(C > 

° = 2 h(C)/,. 2 - C 0 /C l\ h ' C ' _ 2 h(C)^-V C ^ h(c ' 

The expected sample size N required for termination of this test is 
approximately 
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E(N | CEP 2 = C) 




[l-L(C)]ln ^ + L(C)ln 

^)4^)T7 C ^-^ 



The exact operating characteristic and mean sample size functions do not 
differ substantially from the above approximations when a and 3 are 
not below .1 [3]. The variance in sample size, V(N | CEP = C) 
apparently does not admit such good, simple approximations. However 
there is empirical evidence [3] that the standard deviation of N tends 
to be of the order of magnitude of the mean of N. This means that, 
occasionally, sampling may continue substantially beyond E(N). On the 
other hand, it is known [4] that the SPRT achieves the smallest average 
sample size of any test of these hypotheses operating with the same error 
rates a and 3. 
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III. PERFORMING THE TESTS. 



We remarked above that, for the proportion test, the acceptance 

number and rejection number depend on C Q and C-j only through the ratio 

Cq/C-| . This implies that performance of the proportion test, as well as 

its operating characteristic and average sample size functions, depend 

upon the hypothesized values only through Cq/C-j . A similar remark can 

2 

be made for the Rayleigh test inasmuch as CEP may be viewed as a 

2 

scale parameter for the distribution of squared radial miss distance R . 

This means that one can, in effect, measure the squared miss distances 
2 

R.j , and the hypothesized values Cq and C-j in "Cg-units" . In such 

2 2 2 

a case, measurements R ^ are converted to T^ = R./Cq , Cq is taken 
to be Cq = Cq/Cq = 1 , and C-| is converted to cj = C^/Cq. Such a 
point of view allows one to use tabulations of a number of "standard" 
tests in order to perform tests of a much wider variety of hypotheses. 

To illustrate this point, we consider one example of each of the 
tests described above. For the case C-j/Cq = 2 and a = 6 = .1 , 
we present graphical aids which would simplify performance of the tests, 
as well as graphs of the operating characteristic curves and expected 
sample size curves. For the Rayleigh case, we also show a sample size 
variance curve obtained through simulation. 

A. Example Sequential Rayleigh Test. Suppose we wish to test 
H n : CEP 2 <1000 vs. H : CEP 2 > 2000 with ct = 3 = .10 . With 
Ci /C q = 2 , we may take Cq = 1 and C-| = 2 as discussed above, with 
the agreement that the squared radial miss distances actually observed 
are transformed by dividing by 1000 before computing the test statistic 
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n 9 , \ 

E R- . Equivalently, (and more simply), we can compute the test 
i=l 1 

statistic with the "raw" data actually observed and transform the acceptance 

and rejection numbers by multiplying by 1000. Thus the test continues so 
2 

long as E R. is between 

a n = 1000 [ na - 1 n 9] / %a = 2000n - 6339.85 

r =1000 [na + ln9] / %a = 2000n + 6339.85 
n 

Plotted as functions of n , these bounds are parallel lines (as shown 
below) having slope 2000 and intercepts + 6339.85, 




2 

A hypothetical sequence of observed values of E is shown plotted 

against n (shown as dots in the figure); that is, the points (1, y-| ) , 
(2, y-j + y^) »•••* (6, E y^ ) . In performing the test, these points 
would have been sequentially plotted on a previously prepared "control 
chart" with the parallel a n and r p lines. Note that the hypothetical 
test terminated with the 6th observation, since the walk (n, E R^) 
left the continuation region for the first time at n = 6; the terminal 
decision was to accept Hq with the example shown. The operating 
characteristic curve, expected sample size curve and variance in sample 
size curve for this test are shown in figure 2, as functions of C/Cq . 
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Note that the operating characteristic curve passes through (or nearly 
so) the points (a, L(l)) and (1-3, L(2)), as should be the case.* Note 
also that mean sample size is largest for C/Cq values between 1 and 
2, again as should be the case, because such values would typically give 
samples which are not more compatible with one of the hypotheses than the other 
so no early decision is reached. Similar observation can be made with 
respect to variance in sample size. 

B. Example Sequential Proportion Test. The sequential test based 
upon "hit - miss" data can also be conveniently performed with a control 
chart. With the values of (a, 3) Cq and proposed in the preceding 

example, we have for this test the hypotheses Hq : p ^ .5000 vs. H g : p £ p^ = 

1 - 1 / /? = .2929. The acceptance and rejection numbers in this case are 



a n = [_i n9 _ na /2] / ln[/2 - 1] = .3932n + 2.4929 
r n = [1 n9 - na/2] / ln[/2 - 1] - .3932n - 2.4929 



which as functions of n are lines with slope .3932 and intercepts + 
2.4929. In this case the control chart is as shown in figure 3. 




n 



Figure 3 



*The 0C curve does not pass precisely through these points because of 
the Wald approximation of B and A . These curves are based on empirical 
values determined through simulation by Gavlak [3]. 



E(N) V(N) L (C/C 




o 

o 

C\J 



o 

LD 



O 

o o 

r— LD 




12 



Figure 2. Characteristics of Example Figure 4. Approximate Characteristics 

Sequential Rayleigh Test. of Example Sequential 

Proportion Test. 



In this figure we show a hypothetical sequence of values (n, 
which led to acceptance of Hq at the tenth observation. The O.C. 
curve and expected sample size curve are shown in figure 4, as functions 
of C/Cg. These curves are based on Wald approximations. 

It is interesting to compare the O.C. curves and expected sample size 
(E(N)) curves for the Rayleigh and proportion tests, for the example case. 

Note that the O.C. curves are quite similar, but that the E(N) curve 
for the proportion test is roughly twice as high as that for the Rayleigh 
test. For example under Hq the sequential proportion test has Eg(N) *« 18 
(a nonsequential proportion test requires n = 97), whereas the Rayleigh 
has Eg(N) = 8.8 ; under the alternative hypothesis the proportion test 
has E,(N) « 20 whereas the Rayleigh has E^(N) = 10 ; the maximal 

a a 

expected sample size for the proportion test is approximately 26 at C = 1.4Cg 
whereas it is 15 at C = 1 . 5Cq for the Rayleigh. One might interpret these 
comparisons as the "cost" of using the "nonparametric "procedure if the para- 
metric procedure were appropriate. Since these relationships are probably 
typical, we may assert that the "nonparametric" procedure gives protection 
against invalid testing due to departures from the parametric assumption, at 
the expense of additional observations (on the average). 

As is typically true of tests of this type, the expected sample sizes 
will generally decrease with any of the following: 

a. increase in a 

b. increase in 8 

★ 

c. increase in the ratio C-j/Cq • 



A decrease in E(N) for true C values not near the mid point of (Cg,C^) 
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IV. CONCLUSIONS. 



The tests discussed above should prove useful in testing whether 
CEP specifications are being met under various test conditions. The 
"nonparametric" test, the sequential proportion test, requires more 
observations (on the average) than its parametric competitor, but is 
not invalidated by departures from the parametric assumptions required 
by the latter. Indeed, with the sequential proportion test, one does 
not have to be concerned with most of the "model validation" problems 
usually encountered in performing a statistical analysis. (All that is 
required is that the observations be independent, and that the test 
conditions be reasonably homogeneous. 

Research is currently being conducted on a third sequential test 
which may prove useful in the context considered here. This test will 
be based on sequentially observed sample medians, with the null hypothesis 
being rejected when the sample median falls too far from . A second 

area of research effort is concerned with methodology for combining 
results observed under the various test conditions into an overall test of 
performance. Results of these efforts will be reported in a future paper. 
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